VMIproc Runs 81-87¶

27/11/20, updated 01/12/20 with masking + renorm fix PH

Data analysis for 450/440eV pump-probe, 1$\mu$m focus (I think) - should be comparable to previous analysis posted by TMO team (Taran...?), running from v4 preprocessed data.

Using VMIproc class for VMI data analysis & interactive plotting, including uncertainties.

Implemented:

Plotting
Inversion
Uncertainties (bootstrap protocol)
Masking

To do:

Covariance

Background:

# # Memory profiler - OPTIONAL for testing
# # https://timothymonteath.com/articles/monitoring_memory_usage/
# %load_ext memory_profiler
# %memit


# # Quick hack for slow internet connection!
# %autosave 36000

# Standard imports
from pathlib import Path
import sys

import numpy as np
import xarray as xr

# Import class from file
sys.path.append(r'/cds/home/p/phockett/dev/tmo-dev/tmo')
import tmoDataBase as tb
# from vmi import VMI  # VMI class
from inversion import VMIproc  # VMI processing class

tb.setPlotDefaults()  # Set plot defaults (optional)

Load data¶

Set dataset¶

# Set main data path
baseDir = Path('/reg/data/ana15/tmo/tmolw0618/scratch/preproc/v5')

# Setup class object and which runs to read.
data = VMIproc(fileBase = baseDir, runList = range(81,87+1), fileSchema='_v5')

# The class has a bunch of dictionaries holding info on the data
# data.runs['files']

# Read in the files with h5py
# There is very basic checking implemented currently, just to see if 'energies' is present.
data.readFiles()

Read 7 files.
Good datasets: [81, 82, 83, 84, 85, 86, 87]
Invalid datasets: []

Filtering¶

# Add a filter to an existing filterset
# Set energy filter to match previous analysis
# TODO: implement event count filter here to check for saturation
# data.setFilter({'signal':{'energies':[0.05, 0.1]}})
data.setFilter(reset=True)

data.setFilter({'signal':{'evrs':[1,1,70]}})  # For v4 data, gas == evrs[:,70] true/false
data.setFilter({'bg':{'evrs':[0,0,70]}})
data.setFilter({'energies':[0.05, 0.1]})
data.filters

{'signal': {'evrs': [1, 1, 70], 'energies': [0.05, 0.1]},
 'bg': {'evrs': [0, 0, 70], 'energies': [0.05, 0.1]}}

Generate VMI images¶

Note this defaults to electron coords ('xc','yc').

data.genVMIXmulti()

Generating VMI images for filters: signal
Generating VMI images for filters: bg

# Check some metrics
print(data.data[81]['signal']['metrics'])
print(data.data[81]['bg']['metrics'])

{'shots': (73658, 2000), 'selected': 63130, 'events': 126260000, 'norm': 73658}
{'shots': (73658, 2000), 'selected': 10528, 'events': 21056000, 'norm': 73658}

# Check an image with Xarray plotter (Matplotlib)
data.imgStack['signal'].sel(run=81).pipe(np.log10).plot.imshow()

/cds/home/p/phockett/.conda/envs/ps-4.0.8-local-clone/lib/python3.7/site-packages/xarray/core/computation.py:700: RuntimeWarning: divide by zero encountered in log10
  result_data = func(*input_data)

<matplotlib.image.AxesImage at 0x7f7f7e54bfd0>

# Subtraction, and set as new image set
data.imgStack['sub'] = data.imgStack['signal'] - data.imgStack['bg']
data.imgStack

<xarray.Dataset>
Dimensions:  (run: 7, xc: 1048, yc: 1048)
Coordinates:
  * xc       (xc) float64 -0.5 0.5 1.5 2.5 ... 1.044e+03 1.046e+03 1.046e+03
  * yc       (yc) float64 -0.5 0.5 1.5 2.5 ... 1.044e+03 1.046e+03 1.046e+03
  * run      (run) int64 81 82 83 84 85 86 87
    norm     (run) int64 73658 75402 72980 74351 73406 92623 79525
Data variables:
    signal   (xc, yc, run) float64 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    bg       (xc, yc, run) float64 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    sub      (xc, yc, run) float64 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0

array([-5.0000e-01,  5.0000e-01,  1.5000e+00, ...,  1.0445e+03,  1.0455e+03,
        1.0465e+03])

array([-5.0000e-01,  5.0000e-01,  1.5000e+00, ...,  1.0445e+03,  1.0455e+03,
        1.0465e+03])

array([81, 82, 83, 84, 85, 86, 87])

array([73658, 75402, 72980, 74351, 73406, 92623, 79525])

array([[[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
...
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]]])

array([[[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
...
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]]])

array([[[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
...
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]]])

# View full image stack
data.showImgSet()

WARNING:param.RasterPlot16226: Due to internal constraints, when aspect and width/height is set, the bokeh backend uses those values as frame_width/frame_height instead. This ensures the aspect is respected, but means that the plot might be slightly larger than anticipated. Set the frame_width/frame_height explicitly to suppress this warning.

Set centre point¶

This is set as native pixel value, or assumed to be the max point in the image.

# data.setCentre([553, 546])  # Set explictly
data.setCentre()  # Default case - set as max ind.
data.checkCentre()

WARNING:param.OverlayPlot47307: Due to internal constraints, when aspect and width/height is set, the bokeh backend uses those values as frame_width/frame_height instead. This ensures the aspect is respected, but means that the plot might be slightly larger than anticipated. Set the frame_width/frame_height explicitly to suppress this warning.

# All the coord parameters are set in `centreInds`
# Note here that `cList` gives the array indicies for the centre point, based on the pixel coords set.
data.centreInds

{'input': [552.5, 547.5],
 'dims': ['yc', 'xc'],
 'yc': {'coord': array(552.5), 'index': array([553])},
 'xc': {'coord': array(547.5), 'index': array([548])},
 'cList': [553, 548],
 'dMax': [494.0, 499.0]}

Basic processing¶

See the previous demo for info, here we'll just generate subtracted images.

# Subtraction, and set as new image set
data.imgStack['sub'] = data.imgStack['signal'] - data.imgStack['bg']
data.imgStack

<xarray.Dataset>
Dimensions:  (run: 7, xc: 1048, yc: 1048)
Coordinates:
  * xc       (xc) float64 -0.5 0.5 1.5 2.5 ... 1.044e+03 1.046e+03 1.046e+03
  * yc       (yc) float64 -0.5 0.5 1.5 2.5 ... 1.044e+03 1.046e+03 1.046e+03
  * run      (run) int64 81 82 83 84 85 86 87
    norm     (run) int64 73658 75402 72980 74351 73406 92623 79525
Data variables:
    signal   (xc, yc, run) float64 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    bg       (xc, yc, run) float64 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    sub      (xc, yc, run) float64 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0

array([-5.0000e-01,  5.0000e-01,  1.5000e+00, ...,  1.0445e+03,  1.0455e+03,
        1.0465e+03])

array([-5.0000e-01,  5.0000e-01,  1.5000e+00, ...,  1.0445e+03,  1.0455e+03,
        1.0465e+03])

array([81, 82, 83, 84, 85, 86, 87])

array([73658, 75402, 72980, 74351, 73406, 92623, 79525])

array([[[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
...
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]]])

array([[[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
...
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]]])

array([[[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
...
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]],

       [[0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.],
        [0., 0., 0., ..., 0., 0., 0.]]])

Image inversion¶

Multiple methods to be supported... so far only Elio's cpBasex is implemented, and this needs a bit of setup.

# For cpbasex, set the basis path
basisPath = r'/reg/d/psdm/tmo/tmolw0618/scratch/results/tmo_ana/calc/G_r512_k128_l4.h5'

# Set also local version - pip installed version not working with loadG in testing (version mismatch issue, or something in import routine?)
pbasexPath = r'/reg/d/psdm/tmo/tmolw0618/results/modules/pbasex' #/pbasex'

# Import module + basis functions
data.setMethod(pbasexPath = pbasexPath, basisPath = basisPath)

cpBasex import OK, checking for basis functions...
Found basis at /reg/d/psdm/tmo/tmolw0618/scratch/results/tmo_ana/calc/G_r512_k128_l4.h5.

Invert a set of images by setting filterSet (if not set, all images in self.imgStack will be processed). Optionally, set normalisation parameters for the images, the default is norm={'type':'max', 'scope':'global'}, which normalises to the global maximum (i.e. over all images).

Currently the options are:

type:

max
sum
raw

scope:

global
local

# Invert a dataset
filterSet = 'sub'
data.inv(filterSet=filterSet)

/reg/d/psdm/tmo/tmolw0618/results/modules/pbasex/pbasex.py:31: RuntimeWarning: invalid value encountered in true_divide
  betas = IEB[:,nim:].reshape(nx,nl-1,nim)/IE[:,None,:]

# Full results are stored in a dictionay, self.proc[filterSet]
# filterSet = 'signal'
data.proc[filterSet].keys()

dict_keys(['fold', 'pbasex', 'xr'])

# ... where 'xr' contains the results in an Xarray
data.proc[filterSet]['xr']

<xarray.DataArray 'IE' (BLM: 3, E: 512, run: 7)>
array([[[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.85542143,  1.18625065,  1.35658582, ...,  1.86054249,
          1.87010694,  2.26394807],
        [ 1.584021  ,  2.1825303 ,  2.49086432, ...,  3.39562632,
          3.40398422,  4.11860754],
        ...,
        [ 0.02192393,  0.02763356,  0.02935417, ...,  0.04305293,
          0.0492009 ,  0.06955178],
        [ 0.02293437,  0.02926899,  0.0306888 , ...,  0.04503984,
          0.05134804,  0.07291196],
        [ 0.02080901,  0.02674488,  0.02783416, ...,  0.04086939,
          0.04652817,  0.06624388]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [-0.715888  , -0.77826999, -0.82973024, ..., -0.89410312,
         -0.97330106, -0.89292027],
        [-0.7049096 , -0.77057544, -0.82194976, ..., -0.89197449,
         -0.97315639, -0.89267838],
...
        [ 0.66048202,  0.73289361,  0.76940053, ...,  0.80619312,
          0.82506544,  0.85819444],
        [ 0.82225295,  0.8468029 ,  0.93491667, ...,  0.96171003,
          0.98558588,  1.00780313],
        [ 0.91236225,  0.90757257,  1.02694784, ...,  1.04782472,
          1.07461533,  1.08992386]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [ 0.42862268,  0.44782799,  0.57983295, ...,  0.62996665,
          0.75070599,  0.6628786 ],
        [ 0.43527086,  0.45635701,  0.58769118, ...,  0.63959001,
          0.76076219,  0.67379061],
        ...,
        [-0.39927194, -0.42682374, -0.37422606, ..., -0.42094869,
         -0.41788483, -0.42413014],
        [-0.54264574, -0.54728369, -0.52934539, ..., -0.56243639,
         -0.53253836, -0.56463216],
        [-0.6223992 , -0.61205552, -0.61605184, ..., -0.6409392 ,
         -0.59555741, -0.64212169]]])
Coordinates:
  * BLM      (BLM) int64 0 2 4
  * E        (E) float64 0.0 0.000359 0.001436 0.003231 ... 93.01 93.38 93.74
  * run      (run) int64 81 82 83 84 85 86 87
    XS       (E, run) float64 0.0 0.0 0.0 0.0 0.0 ... 2.239 2.707 3.082 4.388
    pixel    (E) int64 0 1 2 3 4 5 6 7 8 ... 503 504 505 506 507 508 509 510 511
    mask     (E, run) bool False False False False False ... True True True True
    norm     (run) float64 38.14 53.41 58.2 62.76 66.24 54.13 53.62
Attributes:
    normType:  {'type': 'max', 'scope': 'global'}

array([[[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.85542143,  1.18625065,  1.35658582, ...,  1.86054249,
          1.87010694,  2.26394807],
        [ 1.584021  ,  2.1825303 ,  2.49086432, ...,  3.39562632,
          3.40398422,  4.11860754],
        ...,
        [ 0.02192393,  0.02763356,  0.02935417, ...,  0.04305293,
          0.0492009 ,  0.06955178],
        [ 0.02293437,  0.02926899,  0.0306888 , ...,  0.04503984,
          0.05134804,  0.07291196],
        [ 0.02080901,  0.02674488,  0.02783416, ...,  0.04086939,
          0.04652817,  0.06624388]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [-0.715888  , -0.77826999, -0.82973024, ..., -0.89410312,
         -0.97330106, -0.89292027],
        [-0.7049096 , -0.77057544, -0.82194976, ..., -0.89197449,
         -0.97315639, -0.89267838],
...
        [ 0.66048202,  0.73289361,  0.76940053, ...,  0.80619312,
          0.82506544,  0.85819444],
        [ 0.82225295,  0.8468029 ,  0.93491667, ...,  0.96171003,
          0.98558588,  1.00780313],
        [ 0.91236225,  0.90757257,  1.02694784, ...,  1.04782472,
          1.07461533,  1.08992386]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [ 0.42862268,  0.44782799,  0.57983295, ...,  0.62996665,
          0.75070599,  0.6628786 ],
        [ 0.43527086,  0.45635701,  0.58769118, ...,  0.63959001,
          0.76076219,  0.67379061],
        ...,
        [-0.39927194, -0.42682374, -0.37422606, ..., -0.42094869,
         -0.41788483, -0.42413014],
        [-0.54264574, -0.54728369, -0.52934539, ..., -0.56243639,
         -0.53253836, -0.56463216],
        [-0.6223992 , -0.61205552, -0.61605184, ..., -0.6409392 ,
         -0.59555741, -0.64212169]]])

array([0, 2, 4])

array([0.000000e+00, 3.590000e-04, 1.436000e-03, ..., 9.301008e+01,
       9.337590e+01, 9.374244e+01])

array([81, 82, 83, 84, 85, 86, 87])

array([[  0.        ,   0.        ,   0.        , ...,   0.        ,
          0.        ,   0.        ],
       [ 56.66368619,  78.57803466,  89.86114949, ..., 123.24357581,
        123.8771315 , 149.96543052],
       [104.92660788, 144.5722634 , 164.99651409, ..., 224.9285531 ,
        225.48218576, 272.81931081],
       ...,
       [  1.45225575,   1.83046577,   1.94443966, ...,   2.85185498,
          3.25910074,   4.60715634],
       [  1.51918801,   1.9387971 ,   2.03284656, ...,   2.98346933,
          3.40132819,   4.82973661],
       [  1.37840275,   1.77159853,   1.84375314, ...,   2.70721533,
          3.08205708,   4.38803884]])

array([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,
        13,  14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,
        26,  27,  28,  29,  30,  31,  32,  33,  34,  35,  36,  37,  38,
        39,  40,  41,  42,  43,  44,  45,  46,  47,  48,  49,  50,  51,
        52,  53,  54,  55,  56,  57,  58,  59,  60,  61,  62,  63,  64,
        65,  66,  67,  68,  69,  70,  71,  72,  73,  74,  75,  76,  77,
        78,  79,  80,  81,  82,  83,  84,  85,  86,  87,  88,  89,  90,
        91,  92,  93,  94,  95,  96,  97,  98,  99, 100, 101, 102, 103,
       104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
       117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129,
       130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
       143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155,
       156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168,
       169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,
       182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,
       195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
       208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220,
       221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
       234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246,
       247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259,
       260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272,
       273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285,
       286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298,
       299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311,
       312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324,
       325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337,
       338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350,
       351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363,
       364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376,
       377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389,
       390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402,
       403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415,
       416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428,
       429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441,
       442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454,
       455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467,
       468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480,
       481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493,
       494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506,
       507, 508, 509, 510, 511])

array([[False, False, False, ..., False, False, False],
       [False, False, False, ..., False, False, False],
       [False, False, False, ..., False, False, False],
       ...,
       [ True,  True,  True, ...,  True,  True,  True],
       [ True,  True,  True, ...,  True,  True,  True],
       [ True,  True,  True, ...,  True,  True,  True]])

array([38.14498052, 53.40962327, 58.19996174, 62.76264432, 66.24066714,
       54.12764627, 53.62134941])

# For rapid plotting per run, use self.plotSpectra()
# Note this defaults to the full data range, aside from a small central mask by default
data.plotSpectra(filterSet = filterSet)

# Set an energy mask and replot
data.setRmask(filterSet, eRange = [0.5,50])

# This can also be set to return a HoloMap for more flexibility
hmap = data.plotSpectra(returnMap = True, filterSet = filterSet)
hmap.overlay('BLM').layout('run').cols(1)

# Here we'll also renormalise to the feature over ~25-30eV

data.renorm(norm={'type':'max', 'scope':'run'}, filterSet=filterSet, eRange=[25,30])

hmap = data.plotSpectra(returnMap = True, filterSet = filterSet, useMask=False, ePlot=[0.1,50])
hmap.overlay('run').layout('BLM').cols(1)

Inversion details¶

Various other stages and results are also logged to the data strucure...

Images are set to 'filterSet-TYPE', where TYPE is:

'fold': Quadrant filter result if quadrant filter is set. These are useful for checking centre point is OK. (TODO: implement quadrant.unfoldQuadrant for full symmetrized images.)
'fit': Reconstructed/fitted images.
'inv': Inverted images

These can be accessed and plotted with the usual functionality.

NOTE: there is currently no default radial masking set, so the default plot scaling will likely be off - adjust with the histogram.

# Quad filter 
data.showImg(name=filterSet + '-fold')

WARNING:param.RasterPlot58937: Due to internal constraints, when aspect and width/height is set, the bokeh backend uses those values as frame_width/frame_height instead. This ensures the aspect is respected, but means that the plot might be slightly larger than anticipated. Set the frame_width/frame_height explicitly to suppress this warning.

data.showImg(name=filterSet + '-inv')

WARNING:param.RasterPlot59637: Due to internal constraints, when aspect and width/height is set, the bokeh backend uses those values as frame_width/frame_height instead. This ensures the aspect is respected, but means that the plot might be slightly larger than anticipated. Set the frame_width/frame_height explicitly to suppress this warning.

# Raw cpbasex output data
data.proc['sub']['pbasex'].keys()

dict_keys(['E', 'IE', 'betas', 'c', 'fit', 'inv'])

Uncertainties¶

A basis Poissonian bootstrapping routine is implemented for estimating uncertainties. This operates on the event data, and allows error propagation through the image generation & analysis, but is (consequently) a little slow. (For more on the method employed, see the wikipedeia page#Poisson_bootstrap).)

Note this routine is a little crude at the moment! It will run image generation for all filterSets, run image subtraction, then image inversion & stats for all filterSets, or just the listed element.

# Run bootstrapping & inversion routine, 5 iterations, and calculate stats for the subtracted data only.
data.bootstrapInv(N = 5, filterSet = 'sub')

*** Bootstrap run for N=5, lambda=1.0, filterSet=sub
Running set 1 of 5
Generating VMI images for filters: signal
Generating VMI images for filters: bg

/reg/d/psdm/tmo/tmolw0618/results/modules/pbasex/pbasex.py:31: RuntimeWarning: invalid value encountered in true_divide
  betas = IEB[:,nim:].reshape(nx,nl-1,nim)/IE[:,None,:]

Running set 2 of 5
Generating VMI images for filters: signal
Generating VMI images for filters: bg

/reg/d/psdm/tmo/tmolw0618/results/modules/pbasex/pbasex.py:31: RuntimeWarning: invalid value encountered in true_divide
  betas = IEB[:,nim:].reshape(nx,nl-1,nim)/IE[:,None,:]

Running set 3 of 5
Generating VMI images for filters: signal
Generating VMI images for filters: bg

/reg/d/psdm/tmo/tmolw0618/results/modules/pbasex/pbasex.py:31: RuntimeWarning: invalid value encountered in true_divide
  betas = IEB[:,nim:].reshape(nx,nl-1,nim)/IE[:,None,:]

Running set 4 of 5
Generating VMI images for filters: signal
Generating VMI images for filters: bg

/reg/d/psdm/tmo/tmolw0618/results/modules/pbasex/pbasex.py:31: RuntimeWarning: invalid value encountered in true_divide
  betas = IEB[:,nim:].reshape(nx,nl-1,nim)/IE[:,None,:]

Running set 5 of 5
Generating VMI images for filters: signal
Generating VMI images for filters: bg

/reg/d/psdm/tmo/tmolw0618/results/modules/pbasex/pbasex.py:31: RuntimeWarning: invalid value encountered in true_divide
  betas = IEB[:,nim:].reshape(nx,nl-1,nim)/IE[:,None,:]

# Stats are output to a dataset, with variables per iteration
data.stats

<xarray.Dataset>
Dimensions:  (BLM: 3, E: 512, run: 7)
Coordinates:
  * BLM      (BLM) int64 0 2 4
  * E        (E) float64 0.0 0.000359 0.001436 0.003231 ... 93.01 93.38 93.74
  * run      (run) int64 81 82 83 84 85 86 87
    XS       (E, run) float64 0.0 0.0 0.0 0.0 0.0 ... 2.271 2.727 3.085 4.376
    pixel    (E) int64 0 1 2 3 4 5 6 7 8 ... 503 504 505 506 507 508 509 510 511
    mask     (E, run) bool False False False False False ... True True True True
    norm     (run) float64 38.63 54.49 57.52 63.13 63.4 52.59 53.09
Data variables:
    0        (BLM, E, run) float64 0.0 0.0 0.0 0.0 ... -0.6433 -0.6386 -0.6494
    1        (BLM, E, run) float64 0.0 0.0 0.0 0.0 ... -0.659 -0.5828 -0.6737
    2        (BLM, E, run) float64 0.0 0.0 0.0 0.0 ... -0.6514 -0.6248 -0.6592
    3        (BLM, E, run) float64 0.0 0.0 0.0 0.0 ... -0.6295 -0.6071 -0.6531
    4        (BLM, E, run) float64 0.0 0.0 0.0 0.0 ... -0.6353 -0.5913 -0.6556

array([0, 2, 4])

array([0.000000e+00, 3.590000e-04, 1.436000e-03, ..., 9.301008e+01,
       9.337590e+01, 9.374244e+01])

array([81, 82, 83, 84, 85, 86, 87])

array([[  0.        ,   0.        ,   0.        , ...,   0.        ,
          0.        ,   0.        ],
       [ 57.38782878,  79.81711053,  89.79180769, ..., 120.78355264,
        122.29118561, 150.85012688],
       [106.20216944, 146.78973729, 164.91945863, ..., 220.59504505,
        222.62904986, 274.381718  ],
       ...,
       [  1.45807902,   1.81618915,   1.96168791, ...,   2.8480019 ,
          3.26502369,   4.61491639],
       [  1.52646096,   1.93223246,   2.0470423 , ...,   2.99620067,
          3.40538424,   4.82415548],
       [  1.38576623,   1.77045525,   1.85476876, ...,   2.72738626,
          3.08464743,   4.37597096]])

array([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,
        13,  14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,
        26,  27,  28,  29,  30,  31,  32,  33,  34,  35,  36,  37,  38,
        39,  40,  41,  42,  43,  44,  45,  46,  47,  48,  49,  50,  51,
        52,  53,  54,  55,  56,  57,  58,  59,  60,  61,  62,  63,  64,
        65,  66,  67,  68,  69,  70,  71,  72,  73,  74,  75,  76,  77,
        78,  79,  80,  81,  82,  83,  84,  85,  86,  87,  88,  89,  90,
        91,  92,  93,  94,  95,  96,  97,  98,  99, 100, 101, 102, 103,
       104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
       117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129,
       130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
       143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155,
       156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168,
       169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,
       182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,
       195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
       208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220,
       221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
       234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246,
       247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259,
       260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272,
       273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285,
       286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298,
       299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311,
       312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324,
       325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337,
       338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350,
       351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363,
       364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376,
       377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389,
       390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402,
       403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415,
       416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428,
       429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441,
       442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454,
       455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467,
       468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480,
       481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493,
       494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506,
       507, 508, 509, 510, 511])

array([[False, False, False, ..., False, False, False],
       [False, False, False, ..., False, False, False],
       [False, False, False, ..., False, False, False],
       ...,
       [ True,  True,  True, ...,  True,  True,  True],
       [ True,  True,  True, ...,  True,  True,  True],
       [ True,  True,  True, ...,  True,  True,  True]])

array([38.63008321, 54.48979429, 57.52096474, 63.13382072, 63.40006591,
       52.58752814, 53.08687796])

array([[[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.90516986,  1.25894365,  1.4162731 , ...,  1.90510137,
          1.92888105,  2.37933707],
        [ 1.67511134,  2.31529313,  2.60125059, ...,  3.4794135 ,
          3.51149556,  4.32778285],
        ...,
        [ 0.02299807,  0.02864649,  0.03094142, ...,  0.04492112,
          0.05149874,  0.0727904 ],
        [ 0.02407665,  0.03047682,  0.0322877 , ...,  0.04725864,
          0.05371263,  0.0760907 ],
        [ 0.02185749,  0.02792513,  0.029255  , ...,  0.04301867,
          0.04865369,  0.06902155]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [-0.7362279 , -0.73680675, -0.83717048, ..., -0.91122477,
         -0.99045156, -0.89575297],
        [-0.72679199, -0.72867247, -0.8283545 , ..., -0.9077713 ,
         -0.99050478, -0.89578523],
...
        [ 0.67956753,  0.71784907,  0.75538647, ...,  0.79557432,
          0.82233351,  0.84210061],
        [ 0.8473952 ,  0.82005099,  0.93103421, ...,  0.95298442,
          0.99206424,  0.99821436],
        [ 0.94059409,  0.87377349,  1.02946001, ...,  1.03969015,
          1.08667501,  1.08472468]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [ 0.40877972,  0.40799988,  0.56838985, ...,  0.63700693,
          0.75358507,  0.70224474],
        [ 0.41336459,  0.41950633,  0.57474421, ...,  0.64599336,
          0.76210095,  0.71264633],
        ...,
        [-0.37490191, -0.44949763, -0.36718526, ..., -0.41708751,
         -0.43456413, -0.43222216],
        [-0.50492596, -0.58534212, -0.53649653, ..., -0.56290614,
         -0.56601062, -0.57195808],
        [-0.57678946, -0.65819966, -0.63167766, ..., -0.64334197,
         -0.63862644, -0.64941599]]])

array([[[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.80104287,  1.14369344,  1.28382308, ...,  1.7945108 ,
          1.78637255,  2.15940473],
        [ 1.48376565,  2.1033714 ,  2.3595901 , ...,  3.2722187 ,
          3.25148654,  3.92758467],
        ...,
        [ 0.0211117 ,  0.02654437,  0.02779114, ...,  0.04136704,
          0.0477981 ,  0.0671356 ],
        [ 0.02214234,  0.0282529 ,  0.02876971, ...,  0.0431127 ,
          0.0498628 ,  0.07036861],
        [ 0.0201231 ,  0.02588879,  0.02594672, ...,  0.03904162,
          0.04516273,  0.06392107]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [-0.7614737 , -0.73914266, -0.84051659, ..., -0.8904834 ,
         -0.97929983, -0.89254185],
        [-0.75071381, -0.73121387, -0.83253696, ..., -0.88886376,
         -0.97932577, -0.89238255],
...
        [ 0.62555433,  0.73429246,  0.75578539, ...,  0.79497202,
          0.81175561,  0.84447604],
        [ 0.76664702,  0.83353516,  0.93132683, ...,  0.9580628 ,
          0.97247245,  1.00299024],
        [ 0.84423272,  0.88539236,  1.03100682, ...,  1.04882122,
          1.06200319,  1.09041668]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [ 0.3560941 ,  0.51859828,  0.56719841, ...,  0.60240262,
          0.74331471,  0.65353485],
        [ 0.36214255,  0.53015484,  0.5744272 , ...,  0.61220915,
          0.7536701 ,  0.66536775],
        ...,
        [-0.39209614, -0.44964493, -0.38911947, ..., -0.41967064,
         -0.4145367 , -0.44218125],
        [-0.49641995, -0.5618177 , -0.55096352, ..., -0.57310958,
         -0.52323946, -0.59128931],
        [-0.55308017, -0.62094006, -0.6429877 , ..., -0.65899686,
         -0.5828336 , -0.67367602]]])

# Similarly to plotSpectra(), there is a plotUncertainties() function for this data
data.plotUncertainties()

# As before, an object can be returned for further plotting options
hmap = data.plotUncertainties(returnMap=True)
hmap.overlay('BLM') #.cols(1)

hmap.overlay('run').layout('BLM').cols(1)

Versions¶

# tmo-dev class version
data.__version__

'0.0.1'

import scooby
scooby.Report(additional=['holoviews', 'xarray'])

Tue Dec 01 18:17:24 2020 PST
OS	Linux	CPU(s)	16	Machine	x86_64
Architecture	64bit	RAM	125.6 GB	Environment	Jupyter
Python 3.7.8 \| packaged by conda-forge \| (default, Jul 31 2020, 02:25:08) [GCC 7.5.0]
holoviews	1.13.5	xarray	0.16.1	numpy	1.19.4
scipy	1.5.3	IPython	7.19.0	matplotlib	3.3.2
scooby	0.5.6